Interactive cooking application

ABSTRACT

Methods, systems and apparatuses, including computer programs encoded on computer storage media, are provided herein for an interactive, educational cooking experience. The embodiments may display a cooking interface that includes various ingredients, kitchen appliances, cookware, and kitchen utensils that may be manipulated by a user. A number of instructions relating to one or more steps of a selected recipe may also be displayed via the cooking interface, such that the user may attempt to follow the instructions by interacting with the displayed items. The embodiments may determine whether such interactions are in accordance with the instructions and may display relevant feedback to the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims benefit of U.S. provisional patentapplication Ser. No. 62/583,551, titled “Interactive CookingApplication,” filed Nov. 9, 2017, which is incorporated by referenceherein in its entirety.

BACKGROUND

This specification relates generally to educational software. Morespecifically, this specification relates to applications, systems andmethods for providing an interactive, educational cooking experience.

More than ever, Americans are taking initiative to ensure that the foodsthey purchase and consume are healthy. Home cooking has become anincreasingly desirable alternative to dining out or purchasing preparedmeals, particularly among young consumers who view home cooking as notonly a cheaper and healthier alternative, but also as an experiencewhere they can develop skills and showcase their creative abilities. Ina recent study, approximately 72% of Americans reported cooking fournights or more per week. Indeed, more than a third of these respondentsreported that they planned to increase their cooking, with youngconsumers being twice as likely as their older counterparts to make thisresolution. Unfortunately, these younger consumers do not have muchconfidence in their culinary abilities—less than half of consumersbetween ages 18 and 34 consider themselves to be “somewhat good” or“good” at cooking.

Cooking applications may provide an attractive solution. A number ofcurrently available cooking applications allow users to learn thecooking process by following steps in a recipe. However, these programsprovide little, if any, educational information, such as informationabout cooking techniques, food ingredients, kitchen appliances andkitchen utensils. Unfortunately, such educational information isparticularly important in cooking because it allows users to developversatile and holistic culinary skills by understanding each of theingredients used in recipes and/or best-practices for safely usingvarious kitchen appliances and utensils. Moreover, conventional cookingapplications fail to provide accurate, detailed feedback to users.

There thus remains a need for interactive cooking applications thatprovide an educational component to allow users to develop an enthusiasmfor cooking, while learning about various cooking equipment andtechniques. It would be beneficial if such applications were adapted toprovide feedback to users while they interactively follow step-by-steprecipe instructions.

SUMMARY

In accordance with the foregoing objectives and others, exemplaryapplications, methods and systems are disclosed herein to provide userswith an interactive, educational cooking experience. The disclosedembodiments may guide users through an interactive cooking process,while providing education information. For example, embodiments of theapplication may display information relating to recipes, dishes and/oringredients (e.g., step-by-step instructions, ingredient flavorprofiles, country of origin, common uses, safety information, portionsizes, nutritional information, etc.); kitchen appliances (e.g.,descriptions, operating instructions, best practices, safetyinformation, etc.); and/or various cookware and kitchen utensils (e.g.common uses, safety information, care instructions, cooking techniques,trivia facts, country of origin, etc.).

In one embodiment, a method for providing an interactive, educationalcooking experience is provided. The method may include displaying anynumber of recipes to a user and receiving a recipe selection from theuser. The method may further include displaying, to the user, recipeinstructions associated with the selected recipe, wherein theinstructions typically include a plurality of steps. Generally, suchsteps may be associated with one or more cooking actions relating to oneor more ingredients, cookware items, kitchen appliances and/or kitchenutensils. The method may further include displaying an interactivecooking interface to a user. The cooking interface may comprise anenvironment such as a digital representation of a kitchen including oneor more of: the ingredients, the cookware items, the kitchen appliances,and the kitchen utensils.

The method may also include displaying a current recipe step to the userand then receiving a user action relating to one or more of thedisplayed items in the kitchen environment. For example, the user mayselect an ingredient and drag it into a pan on a stove. As the usercompletes the action, feedback information may be determined and/orprovided to the user.

The method may further include determining whether the user action iscorrect, based on a comparison between the action and the current recipestep. Upon a determination that the user has taken the correct useraction, the method may then display one or more results of the completedstep. The method may then determine whether the current step is the laststep in the recipe. If so, the method may end by, for example,displaying a completed dish to the user. If not, the current step may beset to the next step in the recipe and such step may be displayed to theuser so that they may perform the next user action.

In one embodiment, a computer-implemented method is provided. The methodmay include (A) storing, by a computer, in a memory: a plurality ofavailable ingredients, one or more available kitchen appliances, one ormore available cookware items, and one or more available kitchenutensils. The method may also include (B) storing, by the computer, inthe memory, a plurality of available recipes, each available recipeassociated with recipe information including a plurality of sequentialrecipe steps, wherein each of the recipe steps relating to a givenavailable recipe may be associated with: (1) one or more requiredcooking items selected from the group consisting of: a requiredingredient selected from the plurality of available ingredients, arequired kitchen appliance selected from the one or more availablekitchen appliances, a required cookware item selected from the one ormore available cookware items, and a required kitchen utensil selectedfrom the one or more available kitchen utensils; (2) a required actionrelating to the one or more required cooking items; and/or (3) aninstruction relating to the required action. The method may include (C)displaying, by the computer, to a user, the plurality of availablerecipes; (D) receiving, by the computer, from the user, a selectedrecipe from the plurality of available recipes; (E) displaying, by thecomputer, to the user, a cooking interface that includes the one or morerequired cooking items associated with a current recipe step, whereineach of the one or more required cooking items may be associated with acurrent state, and wherein each of the one or more required cookingitems may be displayed via the cooking interface according to therespective current state; (F) providing, by the computer, to the user,the instruction associated with the current recipe step; (G) receiving,by the computer, from the user, a user action relating to the one ormore required cooking items associated with the current recipe step; and(H) providing, by the computer, to the user, feedback informationrelating to the user action. In certain embodiments, the method may alsoinclude, (I) upon determining by the computer that the user actionmatches the required action associated with the current recipe step, (1)updating the current state of the one or more required cooking itemsassociated with the current step, based on the user action; and (2) upondetermining by the computer that the current recipe step is not a lastrecipe step associated with the selected recipe, (i) updating thecurrent recipe step to a next recipe step associated with the selectedrecipe, and (ii) repeating steps E through I.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of an exemplary interactive educationalcooking system 100 according to an embodiment.

FIG. 2 shows a block diagram of an exemplary computing machine 200 andmodules 250 according to an embodiment.

FIG. 3 shows an exemplary recipe selection screen 300 according to anembodiment.

FIGS. 4-11 show exemplary interactive cooking interface screens, whereineach screen comprises a notification panel displaying a current step ofa selected recipe and feedback relating to the current step and/or aprevious step of the recipe.

FIG. 12 shows an exemplary recipe completion screen 1200 according to anembodiment.

FIG. 13 shows an exemplary interactive cooking method 1300 according toan embodiment.

DETAILED DESCRIPTION

Various interactive, educational cooking applications, systems andmethods are disclosed herein. Exemplary embodiments may display detailedinstructions relating to any number of recipes that may be digitally“cooked” by a user. The embodiments may provide an interactive cookinginterface comprising any number of selectable ingredients, cookware andbakeware items (collectively referred to herein as “cookware” items),kitchen appliances, and/or kitchen utensils. Each of these displayeditems may be manipulated by the user (e.g., via touch input such asdragging, dropping, rotating, shaking, etc.). The disclosed embodimentsmay receive input from a user (i.e., user action(s)) relating to one ormore of the displayed items, and may provide textual, graphical and/orauditory feedback to the user, based on the user action(s) and/or therecipe instructions.

In certain embodiments, the cooking applications may provide educationalinformation to a user, such as information about dishes, recipes,ingredients, cookware, kitchen appliances, and/or kitchen utensils. Forexample, the interface may provide ingredient information relating toany number of ingredients, as desired by a user or as required by aneducational curriculum. Additionally or alternatively, the cookinginterface may provide positive or negative feedback to the user, basedon received input. For example, the interface may display textual,graphical and/or auditory feedback relating to a status of a dish thatis being prepared by the user (e.g., animations or sounds representingboiling, heating, burning, etc.). As another example, the interface maydisplay textual information confirming that the user has correctlycompleted a recipe step or explaining why a particular user action wasperformed incorrectly.

Referring to FIG. 1, a block diagram of an exemplary system according toan embodiment is illustrated. As shown, the system comprises any numberof users accessing a server 120 via a network 130. In certainembodiments, a user may access the server 120 via a client device 110connected to the network 130.

Generally, a client device 110 may be any device capable of running acooking application and/or of accessing the server 120 (e.g., via thecooking application or via a web browser). Exemplary client devices 110may include general purpose desktop computers, laptop computers,smartphones, and/or tablets. Preferred client devices include those withtouch screens to allow users to view, select and manipulate variousitems displayed by the cooking application (also referred to as a“client application”). In other embodiments, client devices 110 maycomprise virtual reality (“VR”) and/or augmented reality (“AR”) hardwareand software, which allow users to provide input via physical gestures.

The relationship of client 110 and server 120 arises by virtue ofcomputer programs running on the respective computers and having aclient-server relationship to each other. Accordingly, each of theclient devices 110 may have a client application running thereon, wherethe client application may be adapted to communicate with a serverapplication running on a server 120, for example, over a network 130.Thus, the client application and server 120 may be remote from eachother. Such a configuration may allow users of client applications toinput information and/or interact with the server from any location.

As discussed in detail below, a cooking application may be adapted topresent various user interfaces to users. Such user interfaces may bebased on information stored on the client device 110 and/or receivedfrom the server 120. Accordingly, the cooking application may be writtenin any form of programming language, including compiled or interpretedlanguages, or declarative or procedural languages, and it can bedeployed in any form, including as a standalone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. Such software may correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data. For example, a program may include one or more scripts storedin a markup language document; in a single file dedicated to the programin question; or in multiple coordinated files (e.g., files that storeone or more modules, sub programs, or portions of code).

The cooking application can be deployed and/or executed on one or morecomputer machines that are located at one site or distributed acrossmultiple sites and interconnected by a communication network. In oneembodiment, a cooking application may be installed on (or accessed by)one or more client devices 110. For example, a user may download acooking application to their mobile device (e.g., from the Google PlayStore or Apple App Store) or may navigate to a webapp using an internetbrowser.

In certain embodiments, the server 120 and/or the client device 110 maybe adapted to receive, determine, record and/or transmit applicationinformation. The application information may be received from and/ortransmitted to the cooking application. Moreover, any of suchapplication information may be stored in and/or retrieved from one ormore local or remote databases (e.g., database 140).

Exemplary application information may include: user information, recipeinformation, ingredient information, kitchen appliance information,cookware information, kitchen utensil information and/or cooking actioninformation. Generally, user information may include, but is not limitedto: user identification information (e.g., name, screen name, password,image, bio, etc.); contact information (e.g., email address, mailingaddress, phone number, etc.); billing information (e.g., credit cardinformation, billing address, etc.); account balance information (e.g.points or online “currency” awarded to the user for completing certaintasks, points or online “currency” the user has purchased); user skilllevel information (e.g., recipes executed successfully/unsuccessfully,average number of tries to successfully execute recipes, specific stepsexecuted successfully/unsuccessfully); and/or settings information (e.g.night mode on/off, sound on/off, music on/off, notifications, etc.).

Exemplary recipe information may include, but is not limited to: arecipe type or category (e.g., appetizer, entree, dessert, spicy, mild,etc.), a description, images, videos, animations, audio files, recipesteps, cooking actions or techniques, ingredients, kitchen appliances,kitchen utensils, cookware items and/or feedback required to provide aninteractive cooking interface for a particular recipe to a user. Incertain embodiments, the various recipe information may be associatedwith one or more parameters, such as a minimum, maximum and/or requiredquantity, time or temperature.

Each ingredient modeled by the system may be associated with ingredientinformation, which may include, but is not limited to ingredientidentification information (e.g., a name, a description, images, videos,animations, audio files); ingredient educational information (e.g., aflavor profile, texture, country of origin, common uses, nutritionalinformation, safe-handling information and/or trivia facts);permitted/current ingredient parameters (e.g., a mass, a volume, athickness or shape, a density, a moisture level or percentage, aspecific heat capacity, a temperature, etc.); and/or one or morepermitted/current preparation status (e.g., dirty, washed, dried,wet/soggy, skinned, peeled, chopped, sliced, diced, minced, whole,marinated, seasoned, etc.). Ingredient information may also includepermission information, such as but not limited to: permitted cookingactions (e.g., washing, peeling, chopping, baking, boiling, freezing,etc.); permitted cookware items; permitted kitchen appliances; and/orpermitted kitchen utensils.

In certain embodiments, ingredient information may be associated withone or more permitted/current cooking statuses or states, which relateto temperatures of an ingredient, whether an ingredient has been cooked,and/or how an ingredient has been cooked. Exemplary permitted/currentcooking statuses may include, but are not limited to: frozen, thawed,raw, undercooked, cooked, rare, medium-rare, medium, well-done, burned,etc. And such cooking statuses may relate to a specific targettemperature and/or a range of target temperatures (e.g., a chickenbreast ingredient may be associated with a permitted status of “cooked”when it's temperature is between 60 and 75° C.).

In one embodiment, system may determine and/or store specific cookingtimes or ranges of cooking times required for an ingredient to reach atarget temperature(s), wherein such time(s) may be based on: (1) any ofthe ingredient parameters associated with a given ingredient and (2) anyof the kitchen appliances, cookware items, cooking methods and/orcooking temperatures employed to virtually cook a particular ingredient.

It will be appreciated that the application may model nearly anyingredient by employing such ingredient information. As an example, achicken breast ingredient may be associated with ingredient informationcomprising: a current preparation status of “cubed and marinated”; acurrent temperature of 20° C. corresponding to a current cooking statusof “thawed”; and a mass of 100 g. This ingredient may be furtherassociated with a permitted cooking status of “cooked,” for which theingredient in its given state would require 8-12 minutes in a pan beingheated by a stovetop set to medium heat to reach a target temperature ofbetween 60 and 75° C.

Exemplary kitchen appliance information for a given appliance mayinclude, but is not limited to: a name; a description; images; videos;animations; audio files; permitted/current status (e.g., on/off,temperature, empty/full, etc.); permitted/current settings (high heat,low heat, defrost, puree, etc.); permitted/current usage time (fiveminutes, until blended, until defrosted, etc.); permitted/currentingredients (e.g., toast for a toaster; coffee, tea and water for acoffee maker; beverages, soups, and food ingredients for a microwave;fruits, vegetables, legumes, and yogurt for a blender, etc.); and/orpermitted/current cookware items. It will be appreciated that theapplication may model nearly any kitchen appliance by employing suchinformation.

Exemplary cookware information may include, but is not limited to: aname, a description, images, videos, animations, audio files, one ormore materials, a cooking rate, a cooking time, conductivity, availablevolume, dimensions, permitted/current ingredients, permitted/currentappliances; permitted/current utensils; and/or permitted/current status(e.g., full, empty, displayed, hidden, open, closed, temperature, etc.).In certain embodiments, the application may model, display and/or allowusers to interact with various cookware items, such as: pots, pans,lids, skillets, baking trays and others.

Exemplary kitchen utensil information may include, but is not limitedto: a name, a description, images, videos, animations, audio files,common uses, trivia facts, country of origin, safety information,permitted/current status (e.g., in-use, clean, dirty, empty, full, new,rusty, etc.), material (e.g., metal, wood, iron, etc.), conductivity,usage time (twenty seconds to open a wine bottle, ten seconds to usetongs to transfer food from one location to another, two minutes to beateggs with a whisk), permitted/current ingredients and/orpermitted/current cookware items. The application may provide any numberof kitchen utensils for use in preparing recipes. For example thekitchen utensils may include: tongs, whisks, forks, spoons, knives,measuring cups, pitchers, spatulas, graters, ladles, can openers, wineopeners, and/or strainers.

Cooking action information may comprise information pertaining to aparticular cooking action or technique, such as boiling, simmering,filleting, frying, dicing, mincing, whipping, tossing, chopping,measuring, stirring, adding or removing a cookware item to/from akitchen appliance, using a kitchen utensil in combination with acookware item or ingredient, setting a parameter of a kitchen appliance,combining ingredients, and/or others. Exemplary cooking actioninformation may include, but is not limited to: a name, a description,reasons for using the action, a location where the cooking action ismost commonly used, foods for which the cooking action is commonly used,a cooking action parameter, etc. Action parameters may include any ofthe above information relating to utensils, cookware, appliances, and/oringredients to which a particular action relates. For example, a“whipping” action may comprise action parameters such as: egg whites andsugar, a whisk, medium-high speed, and a usage time of two minutes orwhenever stiff meringue peaks form, and the finished product may be ameringue.

In one embodiment, a user action comprises one or more of the groupconsisting of: adding a cookware item to a kitchen appliance, removing acookware item from a kitchen appliance, adding an ingredient to akitchen appliance, removing an ingredient from a kitchen appliance,adding an ingredient to a kitchen utensil, removing an ingredient from akitchen utensil, moving a kitchen utensil to an ingredient, adding aningredient to a cookware item, removing an ingredient from a cookwareitem, adding a first ingredient to one or more additional ingredients,setting a parameter of a kitchen appliance, and setting a parameter of akitchen utensil.

In one embodiment, the server 120 may be connected to one or morethird-party systems 150 via the network 130. Third-party systems 150 maystore information in one or more databases that may be accessed by theserver. Exemplary third-party systems may include, but are not limitedto: payment and billing systems, nutritional information systems, socialmedia and messaging systems, photo storage and sharing systems, addressbook systems, to-do list and shopping list management systems and/orcloud-based storage and backup systems. The server 120 may be capable ofretrieving and/or storing information from third-party systems 150, withor without user interaction. Moreover, the server may be capable oftransmitting stored information to third-party systems.

Referring to FIG. 2, a block diagram is provided illustrating acomputing machine 200 and modules 250 in accordance with one or moreembodiments presented herein. The computing machine 200 may correspondto any of the various computers, servers, mobile devices, embeddedsystems, or computing systems presented herein (e.g., the clientdevice(s) 110, server(s) 120, and/or third-party system(s) 150 of FIG.1). The modules 250 may comprise one or more hardware or softwareelements configured to facilitate the computing machine 200 inperforming the various methods and processing functions presentedherein.

The computing machine 200 may comprise all kinds of apparatuses,devices, and machines for processing data, including but not limited to,a programmable processor, a computer, and/or multiple processors orcomputers. As shown, an exemplary computing machine 200 may includevarious internal and/or attached components, such as a processor 210,system bus 270, system memory 220, storage media 240, input/outputinterface 280, and network interface 260 for communicating with anetwork 230.

The computing machine 200 may be implemented as a conventional computersystem, an embedded controller, a laptop, a server, a mobile device, asmartphone, a set-top box, over-the-top content TV (“OTT TV”), InternetProtocol television (“IPTV”), a kiosk, a vehicular information system,one more processors associated with a television, a customized machine,any other hardware platform and/or combinations thereof. Moreover, acomputing machine may be embedded in another device, such as but notlimited to, a mobile telephone, a personal digital assistant (“PDA”), asmartphone, a tablet, a mobile audio or video player, a game console, aGlobal Positioning System (“GPS”) receiver, or a portable storage device(e.g., a universal serial bus (“USB”) flash drive). In some embodiments,the computing machine 200 may be a distributed system configured tofunction using multiple computing machines interconnected via a datanetwork or system bus 270.

The processor 210 may be configured to execute code or instructions toperform the operations and functionality described herein, managerequest flow and address mappings, and to perform calculations andgenerate commands. The processor 210 may be configured to monitor andcontrol the operation of the components in the computing machine 200.The processor 210 may be a general-purpose processor, a processor core,a multiprocessor, a reconfigurable processor, a microcontroller, adigital signal processor (“DSP”), an application specific integratedcircuit (“ASIC”), a graphics processing unit (“GPU”), a fieldprogrammable gate array (“FPGA”), a programmable logic device (“PLD”), acontroller, a state machine, gated logic, discrete hardware components,any other processing unit, or any combination or multiplicity thereof.The processor 210 may be a single processing unit, multiple processingunits, a single processing core, multiple processing cores, specialpurpose processing cores, coprocessors, or any combination thereof. Inaddition to hardware, exemplary apparatuses may comprise code thatcreates an execution environment for the computer program (e.g., codethat constitutes one or more of: processor firmware, a protocol stack, adatabase management system, an operating system, and a combinationthereof). According to certain embodiments, the processor 210 and/orother components of the computing machine 200 may be a virtualizedcomputing machine executing within one or more other computing machines.

The system memory 220 may include non-volatile memories such asread-only memory (“ROM”), programmable read-only memory (“PROM”),erasable programmable read-only memory (“EPROM”), flash memory, or anyother device capable of storing program instructions or data with orwithout applied power. The system memory 220 also may include volatilememories, such as random-access memory (“RAM”), static random-accessmemory (“SRAM”), dynamic random-access memory (“DRAM”), and synchronousdynamic random-access memory (“SDRAM”). Other types of RAM also may beused to implement the system memory. The system memory 220 may beimplemented using a single memory module or multiple memory modules.While the system memory is depicted as being part of the computingmachine 200, one skilled in the art will recognize that the systemmemory may be separate from the computing machine without departing fromthe scope of the subject technology. It should also be appreciated thatthe system memory may include, or operate in conjunction with, anon-volatile storage device such as the storage media 240.

The storage media 240 may include a hard disk, a compact disc read onlymemory (“CD-ROM”), a digital versatile disc (“DVD”), a Blu-ray disc, amagnetic tape, a flash memory, other non-volatile memory device, asolid-state drive (“SSD”), any magnetic storage device, any opticalstorage device, any electrical storage device, any semiconductor storagedevice, any physical-based storage device, any other data storagedevice, or any combination or multiplicity thereof. The storage media240 may store one or more operating systems, application programs andprogram modules such as module, data, or any other information. Thestorage media may be part of, or connected to, the computing machine200. The storage media may also be part of one or more other computingmachines that are in communication with the computing machine such asservers, database servers, cloud storage, network attached storage, andso forth.

The modules 250 may comprise one or more hardware or software elementsconfigured to facilitate the computing machine 200 with performing thevarious methods and processing functions presented herein. The modules250 may include one or more sequences of instructions stored as softwareor firmware in association with the system memory 220, the storage media240, or both. The storage media 240 may therefore represent examples ofmachine or computer readable media on which instructions or code may bestored for execution by the processor. Machine or computer readablemedia may generally refer to any medium or media used to provideinstructions to the processor. Such machine or computer readable mediaassociated with the modules may comprise a computer software product. Itshould be appreciated that a computer software product comprising themodules may also be associated with one or more processes or methods fordelivering the module to the computing machine via the network, anysignal-bearing medium, or any other communication or deliverytechnology. The modules 230 may also comprise hardware circuits orinformation for configuring hardware circuits such as microcode orconfiguration information for an FPGA or other PLD.

The input/output (“I/O”) interface 280 may be configured to couple toone or more external devices, to receive data from the one or moreexternal devices, and to send data to the one or more external devices.Such external devices along with the various internal devices may alsobe known as peripheral devices. The I/O interface 280 may include bothelectrical and physical connections for operably coupling the variousperipheral devices to the computing machine 200 or the processor 210.The I/O interface 280 may be configured to communicate data, addresses,and control signals between the peripheral devices, the computingmachine, or the processor. The I/O interface 280 may be configured toimplement any standard interface, such as small computer systeminterface (“SCSI”), serial-attached SCSI (“SAS”), fiber channel,peripheral component interconnect (“PCP”), PCI express (PCIe), serialbus, parallel bus, advanced technology attachment (“ATA”), serial ATA(“SATA”), universal serial bus (“USB”), Thunderbolt, FireWire, variousvideo buses, and the like. The I/O interface may be configured toimplement only one interface or bus technology. Alternatively, the I/Ointerface may be configured to implement multiple interfaces or bustechnologies. The I/O interface may be configured as part of, all of, orto operate in conjunction with, the system bus 270. The I/O interface280 may include one or more buffers for buffering transmissions betweenone or more external devices, internal devices, the computing machine200, or the processor 210.

The I/O interface 280 may couple the computing machine 200 to variousinput devices including mice, touch-screens, scanners, biometricreaders, electronic digitizers, sensors, receivers, touchpads,trackballs, cameras, microphones, keyboards, any other pointing devices,or any combinations thereof. When coupled to the computing device, suchinput devices may receive input from a user in any form, includingacoustic, speech, visual, or tactile input.

The I/O interface 280 may couple the computing machine 200 to variousoutput devices such that feedback may be provided to a user via any formof sensory feedback (e.g., visual feedback, auditory feedback, ortactile feedback). For example, a computing device can interact with auser by sending documents to and receiving documents from a device thatis used by the user (e.g., by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser). Exemplary output devices may include, but are not limited to,displays, speakers, printers, projectors, tactile feedback devices,automation control, robotic components, actuators, motors, fans,solenoids, valves, pumps, transmitters, signal emitters, lights, and soforth. And exemplary displays include, but are not limited to, one ormore of: projectors, cathode ray tube (“CRT”) monitors, liquid crystaldisplays (“LCD”), light-emitting diode (“LED”) monitors and/or organiclight-emitting diode (“OLED”) monitors.

Embodiments of the subject matter described in this specification can beimplemented in a computing machine 200 that includes one or more of thefollowing components: a backend component (e.g., a data server); amiddleware component (e.g., an application server); a frontend component(e.g., a client computer having a graphical user interface (“GUI”)and/or a web browser through which a user can interact with animplementation of the subject matter described in this specification);and/or combinations thereof. The components of the system can beinterconnected by any form or medium of digital data communication, suchas but not limited to, a communication network.

Accordingly, the computing machine 200 may operate in a networkedenvironment using logical connections through the network interface 260to one or more other systems or computing machines across the network230. The network 230 may include wide area networks (“WAN”), local areanetworks (“LAN”), intranets, the Internet, wireless access networks,wired networks, mobile networks, telephone networks, optical networks,or combinations thereof. The network 230 may be packet switched, circuitswitched, of any topology, and may use any communication protocol.Communication links within the network 230 may involve various digitalor an analog communication media such as fiber optic cables, free-spaceoptics, waveguides, electrical conductors, wireless links, antennas,radio-frequency communications, and so forth.

The processor 210 may be connected to the other elements of thecomputing machine 200 or the various peripherals discussed hereinthrough the system bus 270. It should be appreciated that the system bus270 may be within the processor, outside the processor, or both.According to some embodiments, any of the processor 210, the otherelements of the computing machine 200, or the various peripheralsdiscussed herein may be integrated into a single device such as a systemon chip (“SOC”), system on package (“SOP”), or ASIC device.

Referring to FIG. 3, an exemplary recipe selection screen 300 of acooking application embodiment is illustrated. As shown, the recipeselection screen 300 may display a recipes list 310 comprising anynumber of available recipes (311-313). The recipes list 310 may displayany recipe information associated with each of the displayed recipes(311-313), such as a name and/or an image. Moreover, each of the recipesin the recipe list may comprise a hyperlink such that a user may selecta recipe by touching or clicking.

Upon selecting a recipe (e.g., Sicilian Pasta 313) from the recipe list,the user may be navigated to an interactive cooking interface. Asdiscussed below, the cooking interface may display various ingredients,kitchen appliances, cookware items and/or kitchen utensils associatedwith the selected recipe.

Additionally or alternatively, a user may select a “create your own”option 325 to advance to the interactive cooking interface withoutselecting an available recipe. In such cases, the cooking interface maycomprise some or all available ingredients, kitchen appliances, cookwareitems and/or kitchen utensils to allow the user to learn about and/orexperiment with the various items.

In one embodiment, one or more of the displayed recipes may betemporarily unavailable to a user (i.e., locked). As shown, lockedrecipes (e.g., “Arroz con Pollo” 311) may be displayed with a visualindication 317, such as a lock icon or the like. A user may unlock suchrecipes via purchase from an online store (e.g., Google Play, AppleStore, etc.), completion of one or more other available recipes, and/orby applying digital “currency” or points previously awarded to the user(e.g., for successfully completing tasks or for sharing variouscontent).

In the illustrated embodiment, a shopping cart icon 327 comprises a linkto an online store where a user can purchase and/or download in-appcontent, such as recipes, digital currency, ingredients, cookware items,kitchen appliances and/or kitchen utensils. The online store may beexternal to the application (e.g., Google Play, Apple Store, etc.) ormay be internal to the application. And the recipes and other in-appcontent available through the online store may be created, uploaded,maintained, sponsored and/or removed by any number of corporate orindividual users (e.g., chefs, grocery stores, restaurants, culinaryschools, meal kit providers, etc.). For example, aningredient-and-recipe meal kit service company may create an in-apprecipe that corresponds to one of their meal kits such that theircustomers, upon receiving the meal kit, may download the recipe and usethe cooking application to understand how to cook the meal kit.

The recipe selection screen 300 may comprise one or more links to allowa user to print application content 342 and/or share such content 341.In certain embodiments, content may be shared via one or more socialmedia networks (e.g., Facebook, Instagram, Twitter, Google, etc.) and/ormessaging applications (e.g., email, SMS, WhatsApp, GroupMe, etc.).Exemplary shareable application content may include screenshots ofvarious steps in a cooking process, user accomplishments or “badges,”and/or images of various in-app items (e.g., completed dishes,ingredients, kitchen utensils, kitchen appliances, cookware, etc.).

In certain embodiments, the recipe selection screen 300 (or a separatehome screen) may allow a user to create a new account and/or login to anexisting account via one or more displayed links. For example, a usermay select a “create account” link 315 to navigate to an accountcreation screen. Such screen may display a number of form fields whereinthe user may enter user information to set up a new account (e.g., firstname, last name, email, password, confirm password, etc.). As anotherexample, a user who has successfully created an account may login to theclient application via a “login” link 320. It will be appreciated thataccount creation and/or login activities may implement a third-partyidentity or authentication service to verify the identity of a user(e.g., Facebook, Google, LinkedIn and/or Twitter).

Once a user signs into the application, they may be guided through aprofile setup process. In such cases, a user may be presented with aform comprising any number of questions and/or entry fields. The usermay enter required or desired user information, such as a name, age,email address, billing information, and/or other information.Information provided by the user may be associated with a user profileand stored in remote and/or local storage. Alternatively oradditionally, user information may be requested or required when a userattempts to purchase a product (e.g., for the first time).

In one embodiment, the recipe selection screen 300 may further comprisea link 330 to a settings screen. The user may select this link to viewand adjust settings provided by the application, such as sound settings,text settings, difficulty settings, display settings, and/oraccessibility settings (e.g., zoom, font size, voiceover, etc.). In oneembodiment, the user may be able to select a “read along” accessibilityoption, which may allow for spoken words to be highlighted in theapplication.

In certain embodiments, the user may be able to turn on/off parentalcontrols via, for example, the settings screen. Such controls may beemployed to prevent young children from accessing various features andcontent of the application, such as in-app purchases and/or specificrecipes. When turned on, parental controls may require a user to answerone or more questions correctly in order to access certain aspects ofthe application. Exemplary questions may relate to grammar, spelling,history, cooking trivia, math and/or various other topics. Alternativelyparental controls may require entry of a previously stored password.

Although not shown, exemplary recipe selection screens may comprise anynumber of links to additional screens, such as a tutorial screens, acontact information screen, a profile screen, a logout screen, and/or auser feedback screen.

Referring to FIG. 4, an exemplary interactive cooking interface screen400 of a cooking application embodiment is illustrated. As shown, thisscreen comprises a digital representation of a kitchen or other cookingenvironment that comprises any number of items that may be employed toperform various steps of the selected recipe, including but not limitedto: ingredients (e.g., pasta 481, cauliflower 482, anchovies 483, basil484, tomatoes 485, garlic 486, cheese 487 and water 489); cookware items(e.g., a lid 421, a pot 431 and a pan 432); kitchen appliances (e.g., astove 450); and/or kitchen utensils (e.g., a spatula 423, a cheesegrater 424, a ladle 425, plates 441-446, and a pitcher 449. It will beappreciated that any of the displayed items may be positioned at anylocation in the cooking environment. For example, if the environmentcomprises a digital representation of a kitchen, the items may belocated in/on a shelf, tabletop, cabinet, refrigerator or freezer. Itwill also be appreciated that the location of a given item may berelative to the location of another item (e.g., “next to the eggs,” or“in the bowl that's on the counter”).

In one embodiment, the cooking environment may only include items thatthe user requires to successfully complete a selected recipe. In otherembodiments, the cooking environment may comprise any number ofnon-required items to increase difficulty.

The cooking interface may provide additional information to the userregarding each item, for example, in the form of a tooltip 490, modal orinformation panel. The tooltip 490 may appear when the user hovers over,taps, clicks, double-clicks, or otherwise selects an item, and thetooltip may provide relevant information relating to the selected item(e.g., ingredient information, cookware information, kitchen applianceinformation and/or kitchen utensil information). The tooltip 490 mayadditionally or alternatively provide magnification of the selected itemsuch that additional details can be discerned.

As shown in FIG. 4, the interactive cooking interface further comprisesa notification panel 410 that is generally adapted to display recipeinformation associated with a selected recipe. In one embodiment, thenotification panel may display instructions 411 or other informationrelating to a current step of the selected recipe. Displayedinstructions 411 may comprise one or more cooking actions to be executedby the user, where such actions may relate to one or more foodingredients, kitchen utensils, cookware items, and/or kitchenappliances. The instructions 411 may refer to absolute position(s)and/or relative position(s) of one or more items.

In the illustrated embodiment, the notification panel 410 displaystextual instructions 411 relating to a first step of a selected“Sicilian Pasta” recipe (FIG. 3 at 313). Specifically, the notificationpanel 410 displays the following instructions 411: “Setup cookingapparatus, choose cooking equipment and set one of the stove dials tohigh heat and another stove dial to medium heat.”

Although not shown, the notification panel may display educationalinformation about the current recipe step (e.g., why the step isperformed and/or information about the importance of a particular itemor action). For example the notification panel may explain that “askillet is well-suited for sautéing vegetables,” or “anchovies aredissolved into the cooked tomatoes in order to provide salty and umaminotes for the dish.” In certain embodiments, such educationalinformation may be displayed upon user selection of the notificationpanel or an icon associated therewith.

The current recipe step may remain on the screen as the user isexecuting the corresponding action, or it may be hidden. Optionally, theuser may hide the instruction 411. Additionally or alternatively, theuser may have the option of selecting a link 461 to view the entirerecipe, or to go back to a screen displaying a previous step.

As the user executes the action(s) required by the current recipe step,the cooking interface 400 may provide hints or visual/audio/textualindications in order to guide the user in correctly executing the step.For example, in one embodiment, the cooking interface may display one ormore ingredients, cookware items, kitchen appliances, and/or kitchenutensils relevant to the current recipe step more prominently than thenon-relevant items on the cooking interface. In such embodiment, thecooking interface may flash the relevant item(s). It will be appreciatedthat in other embodiments, the cooking interface may highlight, enlarge,move, and/or otherwise draw attention to the relevant item(s).

In any event, the user may manipulate the displayed items to follow thecurrent step instructions 411 provided by the notification panel 410.For example, in the illustrated embodiment, the user drag and drop thepot 431 and the pan onto burners 451 of a stove 450. The user may alsotap and/or rotate the stove dials 452, 453 to set the temperatures ofthe burners 451. In other embodiments, the user may select and moveitems by highlighting, single-clicking, double-clicking, tapping, etc.

In one embodiment (not shown), the user may be required to perform anaction more than once in a single step, depending upon the portion sizeof the ingredient indicated in the instructions 411. For example, if aninstruction requires the user to measure one teaspoon of salt to pour ina pot, and one teaspoon is the equivalent of three shakes, then a usermay need to drag a salt shaker to a pot, and “shake” it three times(e.g., by shaking their device or by moving the salt back and forth) tocomplete the instruction successfully.

In certain embodiments, the cooking interface may display one or more ofthe items in motion and in use, based on the user's actions. Forexample, the cooking interface may show: a ladle 425 dipped in water andlifted, a spatula 423 moving around in a pan 432 to mix ingredients, asalt dispenser 447 or pepper dispenser 448 flipping upside down withsalt/pepper falling into a pan, tomatoes 485 being emptied from a plate445, water 489 being emptied from a water pitcher 449, a grater 424grating a chunk of cheese 487 with cheese shavings falling into a pan432, etc. In such embodiments, the items in use may further beaccompanied by sounds and/or haptic feedback. For example, in certainembodiments, the action of straining oil into a jar, the shaking of thesalt dispenser, and the squeezing of a lemon may be accompanied by theircorresponding sounds.

In one embodiment, once an item has been selected and moved from itsoriginal position, usage of the item may be depicted by one or more ofthe following: showing a vacant space in its original position(indicating unavailability of the item for further use), and/or showinglower levels of contents in a container (e.g., a lower level of oliveoil in the transparent bottle).

In certain embodiments, ingredients, cookware items, kitchen appliancesand/or kitchen utensils used in a step may be returned to their originalpositions automatically for use in the next recipe step. Alternatively,the application may require the user to manually return items to theiroriginal position.

During the current recipe step, the cooking interface 400 may displayvisual and/or auditory feedback to the user. Such feedback may bepositive or negative and may include: text (e.g., “Wow! That smellsgreat!” or “I don't know, that smells a bit strange . . . ”), sounds,emojis or animations appearing on the screen with positive or negativefacial expressions, visual indications of the cooking process (e.g.smoke/steam from pot, coils on stove turning red from heat, a rawchicken breast turning brown from heat, a dry looking dish, etc.) andothers.

In certain embodiments, the cooking interface 400 may also display otherforms of visual and/or auditory feedback to the user regarding timinginvolved in a particular step. For example, a timer may appear on thescreen during a step (see FIG. 6 at 690). Such timer may beautomatically set, and may emit a sound or a light upon completion ofcooking. In other embodiments, the user may need to manually set thetimer on the screen. In yet other embodiments, such as for an increaseddifficulty setting, the user may receive no indication upon completionof cooking. The cooking interface may also show positive or negativefeedback during a single step and/or once the current step is completed(discussed below in reference to FIG. 5).

The cooking interface 400 may include one or more links 462 to allow auser to take a photo of the current step. Additionally or alternatively,the interface may allow the user to record notes relating to a selectedrecipe and/or a specific step in the recipe. Such notes may be storedlocally and/or may be synced to one or more remote note-takingapplications.

In one embodiment, the cooking interface 400 may comprise one or morelinks 461 to allow the user to view, save and/or print informationrelating to a selected recipe. Upon selecting such link(s), the user maybe presented with a screen, modal or window showing: (1) each steprequired to prepare the selected recipe and/or (2) an itemized list ofall ingredients required to prepare the selected recipe. In oneembodiment, the user may only be able to view and/or print the itemizedlist of all ingredients required to prepare the selected recipe aftersuccessfully completing the dish. The user may then select one or moreof the displayed items in order to print, save, and/or download a listof such items (e.g., for use as a shopping list). In one particularembodiment, the user may select one or more of the displayed items toautomatically add them to a shopping list feature of the cookingapplication and/or to sync such items to another local or cloud-basedshopping list application. Additionally or alternatively, the cookingapplication may allow the user to easily purchase such items, either viaa shopping feature of the cooking application or via a connection to athird-party ecommerce application.

Although not shown, exemplary cooking interface screens may comprise anynumber of links to additional features and screens, such as a link tosave progress, a link to return to a previous screen or move forward tothe next screen, and/or a link to restart/pause/continue theapplication.

Referring to FIG. 5, an exemplary interactive cooking interface screen500 of a cooking application embodiment is illustrated. As shown, thenotification panel 510 displays the current step of the given recipe 511and optional feedback information 512.

In the illustrated embodiment, the user has correctly performed theprevious instructions (FIG. 4 at 411) and has continued on to the nextstep of the selected recipe (i.e., the system has updated the currentstep to the next step in the recipe). As shown, the notification panel510 now displays instructions 511 corresponding to the updated currentstep (i.e., “Step 2: Now boil some water for the pasta.”). Moreover, thecooking interface screen 500 has been updated to reflect the completedprevious step (i.e., the pan 532 and pot 531 are located on the stove550, and the dials 552, 553 are rotated).

In one embodiment, the notification panel displays feedback information512 relating to a previous step of the recipe. For example, if theprevious step was correctly completed, the notification panel may showpositive feedback relating to the previous step (e.g., “Good job! Youcorrectly set the stovetop dials and chose the pot and pan.”). Asanother example, such feedback may comprise a sound (e.g., clapping orspoken words); a happy face emoji; an animation appearing on the screen;and/or checking off or striking through the previous recipe step fromthe recipe steps list. And, as yet another example, no explicit feedbackrelating to the previous step may be displayed.

The feedback information 512 may alternatively relate to the currentstep of the recipe. For example, if the current step is not correctlycompleted, the notification panel may show negative feedback relating tothe current step (e.g., “You did not set the stovetop to medium heat.You must set the stovetop to medium heat before proceeding to step 3.”).As another example, such feedback may comprise a sound (e.g., crying orspoken words); a sad emoji; and/or animation appearing on the screen. Inone particular embodiment, such feedback information may comprise one ormore visual indications relating to the cooking process (e.g.,displaying a “burned” ingredient in a dark color).

In certain embodiments, negative feedback information displayed via thenotification panel 510 may comprise one or more hints relating toincorrect user actions (e.g., “Looks a bit dry. Are you sure you addedthe right amount of water?” or “Check that the right ingredient(s)was/were used.”). In such cases, the user may not be allowed to continueto the next step until they complete the current step correctly. A usermay be allowed a certain number of attempts to retry the incorrectlyperformed step, or alternatively, the user may be allowed an unlimitednumber of attempts to do so. In one embodiment, the cooking applicationmay demonstrate how to perform a given step after a certain number ofincorrect attempts and may then allow the user to continue on to thenext step.

In the illustrated embodiment, the user may interact with the displayeditems to follow the directions and complete the current recipe step 511.For example, the user may select the pitcher 549 and drag it to the pot531 to pour water 589 from the pitcher into the pot. The user may thenwait a predetermined amount of time for the water to boil. The start ofboiling may be indicated by visual indications of the cooking process(i.e., the water in the pan forming bubbles and steam, the sound ofboiling water, a light or beep from the stove, a notification displayedin the notification panel, etc.).

Referring to FIG. 6, an exemplary interactive cooking interface screen600 of a cooking application embodiment is illustrated. As shown, thescreen comprises a notification panel 610 displaying positive feedback(i.e., “Great work! You added water to the pot and now it's boiling.”)612 because the user has correctly completed the previous step (FIG. 5at 511). The notification panel 610 also displays instructions 611relating to the updated current recipe step (i.e., “Step 3: Go ahead andboil the cauliflower in the water for 5 minutes. Also, sauté the garlicand olive oil in a pan.”).

In the illustrated embodiment, the cooking interface screen 600 has beenupdated to reflect the completed previous step. For example, the pot 631is shown on the stove 650 and boiling water 689 is displayed within thepot. Moreover, the pitcher 649 is now shown in an empty state becausethe water was previously emptied into the pot.

To follow the directions 611 for the current recipe step, a user maydrag and drop cauliflower 682 from its position on the tabletop to thepot 631 of boiling water 689 on the stove 650. The user may then dragand drop garlic 686 and olive oil into the pan 632.

In certain embodiments, the cooking interface 600 may display a timer690. As shown, the user may manually set the hours 691, minutes 692,and/or seconds 693 of the timer 690 by, for example, manipulatingbuttons 695, 696 or using a keyboard to input a desired time. In orderto follow the directions 611 in the illustrated embodiment, the user mayset the timer 690 to five minutes.

In certain embodiment, the user may set aside any prepared mixture oringredient during the cooking process, such as: cooked vegetables,sauces, cooked proteins, baked pastries, etc. Generally, the user mayset aside cooked ingredients after a predetermined amount of time haspassed and/or feedback is provided regarding the completion of cooking.

In one embodiment, ingredients may be set aside by utilizing one or moreavailable kitchen utensils to drag and drop the cooked ingredients backto their initial position. For example, the user may pick up a ladle625, then use the ladle to scoop the cauliflower 682 from the pot 631and drop it onto the original cauliflower plate. In other embodiments,the user may move cooked ingredients to a new location, such as an icebath or strainer (not shown), which may be positioned anywhere withinthe cooking interface. And, in yet other embodiments, the user may inputa time and then simply leave the ingredients in a cookware item to moveon to the next step.

Referring to FIG. 7, an exemplary interactive cooking interface screen700 of a cooking application embodiment is illustrated. As shown, thenotification panel 710 of the interface screen displays positivefeedback 712 because the user has correctly completed the previous step(FIG. 6 at 611). The notification panel 710 also displays instructionsrelating to the updated current recipe step 711.

In the illustrated embodiment, the cooking interface screen 700 has beenupdated to reflect the completed previous step. As shown, garlic 786 andolive oil are displayed in the pan 732 on the stove 750, with steam andsmoke 735 emanating therefrom; cauliflower 782 is displayed in the pot731; and both the cauliflower plate 742 and the garlic plate 746 aredisplayed in an empty state.

To follow the displayed instructions, the user may drag and dropanchovies 783 and tomatoes 785 into the pan 732. After a predeterminedamount of time, the cooking interface screen 700 may display visualfeedback to the user that the anchovies and tomatoes are cooked. Forexample, the anchovies 783 may be shown to dissolve, the tomatoes 785may be shown to sweat and shrinking, and steam and smoke may be shownemanating from the pan 732.

Upon receiving the visual feedback, the user may drag the salt 747 andpepper 748 dispensers to the pan and then move such items back and forthto add the respective ingredients. Such action may result in ananimation of salt and pepper crystals falling down onto the cookingmixture of anchovies 783 and tomatoes 785.

Referring to FIG. 8, an exemplary interactive cooking interface screen800 of a cooking application embodiment is illustrated. As shown, thescreen comprises a notification panel 810 that displays feedback 812relating to a correctly completed previous recipe step (FIG. 7 at 711)and instructions relating to an updated current recipe step 811.

In the illustrated embodiment, the cooking interface screen 800 has beenupdated to reflect the completed previous step. For example, theanchovies plate 843 and the tomatoes plate 845 are both shown to beempty. Moreover, the mixture of anchovies, tomatoes, garlic andcauliflower may be shown to be cooked/cooking in the pan 832 via theinclusion of animated steam and/or a change in color.

To complete the displayed instructions, the user may drag and drop thelid 821 from its initial position onto the pan 832. Upon such action,the application may show the lid 821 positioned on top of the pan 832.The application may also display an animation representing dissipationof any steam that was emanating from the pan before the lid was applied.

Referring to FIG. 9, an exemplary interactive cooking interface screen900 of a cooking application embodiment is illustrated. As shown, thescreen comprises a notification panel 910 that displays feedback 912relating to a correctly completed previous recipe step (FIG. 8 at 811)and instructions relating to a current recipe step 911.

In the illustrated embodiment, the cooking interface screen 900 has beenupdated to reflect the completed previous step. Specifically, the lid921 is shown on the pan 932 and the original lid location is now shownto be empty.

To complete the displayed instructions, the user may drag and drop thedry pasta 981 into the pot of boiling water 931 located on the stove950. The user may then wait a certain amount of time for the pasta tocook in the boiling water. Once the user has determined that the pasta981 has finished cooking, the user may set it aside (e.g., by draggingand dropping back to an initial location). Additionally, the user maydrag the lid 921 from the pan 932 back to its initial position, and thentap/rotate the stove dial 953 to set the burner associated with the pan932 to low heat.

Exemplary visual feedback displayed for the cooking actions associatedwith this recipe step may include the: the pasta falling into the pot,water splashing out of the pot, the pasta becoming softer and coiled,and/or the pasta changing in color. Additional visual or auditoryindications may be transmitted when the pasta has finished cooking, whenthe lid is removed from the pan, and/or when the user correctly sets thestove to low heat.

Referring to FIG. 10, an exemplary interactive cooking interface screen1000 of a cooking application embodiment is illustrated. As shown, thescreen comprises a notification panel 1010 that displays feedback 1012relating to a correctly completed previous recipe step (FIG. 9 at 911)and instructions relating to a current recipe step 1011.

In the illustrated embodiment, the cooking interface screen 1000 hasbeen updated to reflect the completed previous step. As shown, thecooked pasta 1071 is displayed on the pasta plate 1041; the lid 1021 isshown to be removed from the pan 1032; the pot 1031 is shown to be emptyand removed from the stove 1050 burner 1051; and the lidless pan 1032 isshown to include the cooked mixture.

Accordingly, to complete the instructions of the current step 1011, theuser may first drag and drop the cooked pasta 1071 into pan 1032. Theuser may then select a kitchen utensil, such as a spatula 1023 or tongs,and move the utensil back and forth within the pan 1032 to mix theingredients. Alternatively, the user may simply drag the utensil to thepan 1032 and the screen may automatically depict the utensil movingaround the pan to mix the ingredients.

The user may then top the dish with cheese 1087 and basil 1084. In thisstep, the user may select the grater 1024 and block of cheese 1087, andmove them both to the pan 1032. Such action may automatically result ina visual depiction of the grater 1024 grating the cheese 1087 over thepan 1032, with grated cheese falling downward onto the mixture.Alternatively, the user may repeatedly drag the grater 1024 along theblock of cheese 1087 to manually grate cheese onto the mixture. The usermay then drag and drop the basil 1084 into the pan 1032.

Referring to FIG. 11, an exemplary interactive cooking interface screen1100 of a cooking application embodiment is illustrated. As shown, thescreen comprises a notification panel 1110 that displays feedback 1112relating to a correctly completed previous recipe step (FIG. 10 at1011). In this embodiment, the notification panel does not display acurrent recipe step, as all of the recipe steps have been successfullycompleted by the user. Rather, the notification panel displays a message1113 indicating successful completion of the recipe.

In the illustrated embodiment, the cooking interface screen 1100 hasbeen updated to reflect the completed previous step. Specifically, thecompleted dish 1175 is displayed in the pan 1132.

Referring to FIG. 12, an exemplary recipe completion screen 1200 of acooking application embodiment is illustrated. In one embodiment, thisscreen may be displayed to the user upon successful completion of allsteps associated with a selected recipe.

As shown, the recipe completion screen 1200 may display a notification1210 indicating successful completion of a recipe. In one embodiment,the completed recipe screen 1200 may further comprise a dish displayarea 1220 wherein a visual depiction of the resulting dish 1221 isdisplayed to the user. The dish display area 1220 may include a numberof links or buttons to allow the user to zoom in or out 1225 and/or torotate 1226 the displayed dish.

The recipe completion screen 1200 may further comprise a recipes list1230 showing any number of recipes 1231-1233 available for selection. Asshown, a visual indication 1235 such as a checkmark icon may bedisplayed alongside a recipe 1233 in the recipes list 1230 to indicatethat the corresponding recipe has been completed by the user. In certainembodiments, the user may be able to select a completed recipe to viewsome or all of the recipe steps.

In one embodiment, an award may be provided to the user upon completionof a recipe. For example, the user may be awarded in-app currency,points, discounts, ingredients, cookware, kitchen appliances and/orkitchen utensils, etc. In one particular embodiment, completion of agiven recipe may result in the unlocking of a previously locked recipe1231 such that the unlocked recipe may be selected by the user. In suchcase, the newly unlocked recipe may be displayed along with a visualindication 1237, such as an unlock icon.

Generally, the recipe completion screen may comprise any number of linksto allow a user to take a photo of the completed dish 1201, print thecorresponding recipe instructions 1202 and/or share recipe content 1203via social media platforms, messaging applications, etc. In certainembodiments, user achievements, recipes, captured images, notes andother content may be shared to an online storage account where suchinformation may be maintained privately or shared with others.

Referring to FIG. 13, an exemplary method 1300 according to anembodiment is illustrated. As shown, the method may begin when a userselects a recipe from a list of available recipes and such selectedrecipe is received by the system 1305.

At step 1310, the system displays an interactive cooking interface basedon the selected recipe. As discussed above, the interface may compriseany number of ingredients, cookware items, kitchen appliances and/orkitchen utensils that may be selected and/or manipulated by the user todigitally “cook” the selected recipe.

At step 1315, the system displays instructions relating to the currentstep of the selected recipe. A selected recipe may be associated with aplurality of steps, where each step comprises instructions. Theinstructions relate to one or more cooking actions, each of which isassociated with one or more of the displayed ingredients, cookwareitems, kitchen appliances, and kitchen utensils.

At step 1320, the user attempts to follow the displayed instructions byperforming a user action and such input is received by the system. Atstep 1325, the system may display an updated cooking interface based onthe user action. The updated interface may comprise an updated status ofeach of the items. For example, the system may display visual feedback,such as an ingredient moving from one location to another, an amount ofan ingredient changing, a color of an item changing, steam, boiling,burning, glowing, browning, etc.

Upon completion of the user action, the system determines that the useraction satisfies the instructions associated with the current recipestep 1330. In certain embodiments, the updated interface may display anotification indicating that the current recipe step has beensuccessfully completed by the user, when the user action satisfies theinstructions associated with the step.

At step 1335, the system determines whether the current step is the laststep in the recipe. If so, the method may end 1350. In certainembodiments, a dish completion screen comprising an image or video ofthe completed dish may be displayed to the user upon completion of therecipe. Optionally, the user may also be presented with a reward forcompleting the recipe, such as but not limited to awarding the userin-app currency, unlocking a recipe and/or awarding the user one or morebadges or stickers.

However, if the current step is not the final step in the selectedrecipe, the method may continue by updating the current step to the nextstep in the recipe 1340 and returning to step 1315. In such cases,updated instructions associated with the next step (i.e., the updatedcurrent step) may be displayed to the user such that they may continuethe interactive cooking experience.

Various embodiments are described in this specification, with referenceto the detailed discussed above, the accompanying drawings, and theclaims. Numerous specific details are described to provide a thoroughunderstanding of various embodiments. However, in certain instances,well-known or conventional details are not described in order to providea concise discussion. The figures are not necessarily to scale, and somefeatures may be exaggerated or minimized to show details of particularcomponents. Therefore, specific structural and functional detailsdisclosed herein are not to be interpreted as limiting, but merely as abasis for the claims and as a representative basis for teaching oneskilled in the art to variously employ the embodiments.

The embodiments described and claimed herein and drawings areillustrative and are not to be construed as limiting the embodiments.The subject matter of this specification is not to be limited in scopeby the specific examples, as these examples are intended asillustrations of several aspects of the embodiments. Any equivalentexamples are intended to be within the scope of the specification.Indeed, various modifications of the disclosed embodiments in additionto those shown and described herein will become apparent to thoseskilled in the art, and such modifications are also intended to fallwithin the scope of the appended claims.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a sub combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

All references including patents, patent applications and publicationscited herein are incorporated herein by reference in their entirety andfor all purposes to the same extent as if each individual publication orpatent or patent application was specifically and individually indicatedto be incorporated by reference in its entirety for all purposes.

What is claimed is:
 1. A computer-implemented method comprising: A.storing, by a computer, in a memory: a plurality of availableingredients; one or more available kitchen appliances; one or moreavailable cookware items; and one or more available kitchen utensils; B.storing, by the computer, in the memory, a plurality of availablerecipes, each available recipe associated with recipe informationcomprising a plurality of sequential recipe steps, wherein each of therecipe steps relating to a given available recipe is associated with:one or more required cooking items selected from the group consistingof: a required ingredient selected from the plurality of availableingredients; a required kitchen appliance selected from the one or moreavailable kitchen appliances; a required cookware item selected from theone or more available cookware items; and a required kitchen utensilselected from the one or more available kitchen utensils; a requiredaction relating to the one or more required cooking items; and aninstruction relating to the required action; C. displaying, by thecomputer, to a user, the plurality of available recipes; D. a receiving,by the computer, from the user, a selected recipe from the plurality ofavailable recipes; E. displaying, by the computer, to the user, acooking interface comprising the one or more required cooking itemsassociated with a current recipe step, wherein each of the one or morerequired cooking items is associated with a current state, and whereineach of the one or more required cooking items is displayed via thecooking interface according to the respective current state; F.providing, by the computer, to the user, the instruction associated withthe current recipe step; G. a receiving, by the computer, via thecooking interface, a user action relating to the one or more requiredcooking items associated with the current recipe step, wherein the useraction comprises one or more of the group consisting of; adding arequired cookware item to a required kitchen appliance, removing arequired cookware item from a required kitchen appliance, adding arequired ingredient to a required kitchen appliance, removing a requiredingredient from a required kitchen appliance, adding a requiredingredient to a required kitchen utensil, removing a required ingredientfrom a required kitchen utensil, moving a required kitchen utensil to arequired ingredient, adding a required ingredient to a required cookwareitem, removing a required ingredient from a required cookware item,adding a first required ingredient to one or more additional requiredingredients, setting a parameter of a required kitchen appliance, andsetting a parameter of a required kitchen utensil; H. providing, by thecomputer, to the user, feedback information relating to the user action;and I. upon determining, by the computer, that the user action matchesthe required action associated with the current recipe step: updatingthe current state of the one or more required cooking items associatedwith the current step, based on the user action; and upon determining,by the computer, that the current recipe step is not a last recipe stepassociated with the selected recipe: updating the current recipe step toa next recipe step associated with the selected recipe; and repeatingsteps E through I wherein the selected recipe is associated with a firstrecipe step that itself is associated with: first required cooking itemscomprising: a first required kitchen appliance associated with aninitial state and a final state, both comprising an active status and alocation within the cooking interface; a first required cookware itemassociated with: an initial state comprising an empty status, a locationwithin the cooking interface that is on or in the first required kitchenappliance: and a final state comprising a full status and the location;and a first required ingredient associated with: an initial statecomprising an uncooked status and an initial location within the cookinginterface: and a final state comprising a cooked status and a finallocation within the cooking interface that is on or in the firstrequired cookware item and that is different than the initial location:a first required action comprising moving the first required ingredientfrom the initial location to the final location: and a first instructionrelating to the first required action, and wherein, when the firstrecipe step is the current recipe step; the first instruction isdisplayed via the cooking interface; until a user action matching thefirst required action is received, each of the first required cookingitems is displayed via the cooking interface according to its initialstate; and upon receiving a user action that matches the first requiredaction, each of the first required cooking items is displayed via thecooking interface according to its final state.
 2. Acomputer-implemented method according to claim 1, wherein the recipeinformation further comprises one or more of the group consisting of: aname, a description, an image, a video, an animations, and an audiofile.
 3. A computer-implemented method according to claim 1, whereineach of the available ingredients stored in the memory is associatedwith ingredient information comprising one or more of the groupconsisting of: a name, a description, an image, a video, an animation,an audio file, a flavor profile, a texture, a country of origin, acommon use, nutritional information, safe-handling information,permitted parameters, a current parameter, a current preparation status,permitted preparation statuses, permitted kitchen appliances, a currentkitchen appliance, permitted cookware items, a current cookware item,permitted kitchen utensils and a current kitchen utensil.
 4. Acomputer-implemented method according to claim 3, wherein the ingredientinformation further comprises one or more permitted cooking states.
 5. Acomputer-implemented method according to claim 4, wherein each of thepermitted cooking states relates to a range of target temperatures.
 6. Acomputer-implemented method according to claim 1, wherein each availablekitchen appliances stored in the memory is associated with applianceinformation comprising one or more of the group consisting of: a name, adescription, an image, a video, an animation, an audio file, permittedtemperature settings, a current temperature setting, permitted usagetimes, a current usage time, permitted ingredients, current ingredients,permitted cookware items, and current cookware items.
 7. Acomputer-implemented method according to claim 1, wherein each of theavailable cookware items stored in the memory is associated withcookware information comprising one or more of the group consisting of:a name, a description, an image, a video, an animation, an audio file,one or more materials, a cooking rate, a conductivity, permittedstatuses, a current status, permitted ingredients, current ingredients,permitted appliances, a current appliance, permitted utensils, and acurrent utensil.
 8. A computer-implemented method according to claim 1,wherein each of the available kitchen utensils stored in the memory isassociated with utensil information comprising one or more of the groupconsisting of: a name, a description, an image, a video, an animation,an audio files, permitted statuses, a current status, permitted usagetimes, a current usage time, permitted ingredients, current ingredients,permitted appliances, a current appliance, permitted cookware items, anda current cookware item.
 9. A computer-implemented method according toclaim 1, wherein the cooking interface further comprises a digitalrepresentation of a kitchen.
 10. A computer-implemented method accordingto claim 1, wherein the user action comprises setting a parameter of arequired kitchen appliance.
 11. A computer-implemented method accordingto claim 10, wherein the user action further comprises adding a requiredingredient to the required kitchen appliance.
 12. A computer-implementedmethod according to claim 11, wherein the parameter of the requiredkitchen appliance comprises one or more of the group consisting of: atime and a temperature.
 13. A computer-implemented method according toclaim 1, wherein the feedback information comprises one or more of:text, video, sound and an animation.
 14. A computer-implemented methodaccording to claim 1, wherein: the one or more required cooking itemsassociated with the current recipe step comprises a required ingredientand a required kitchen appliance; the required ingredient is furtherassociated with a plurality of permitted states, each relating to atarget temperature; the required kitchen appliance is further associatedwith a plurality of permitted states, each relating to a permittedtemperature; and the user action comprises: setting the required kitchenappliance to a permitted temperature; and adding the required ingredientto the required appliance.
 15. A system comprising one or more computersand one or more storage devices storing instructions that, when executedby the one or more computers, cause the one or more computers to performoperations comprising; A. storing, in the one or more storage devices: aplurality of available ingredients; one or more available kitchenappliances; one or more available cookware items; and one or moreavailable kitchen utensils; B. storing, in the one or more storagedevices, a plurality of available recipes, each available recipeassociated with recipe information comprising a plurality of sequentialrecipe steps, wherein each of the recipe steps relating to a givenavailable recipe is associated with: one or more required cooking itemsselected from the group consisting of: a required ingredient selectedfrom the plurality of available ingredients; a required kitchenappliance selected from the one or more available kitchen appliances; arequired cookware item selected from the one or more available cookwareitems; and a required kitchen utensil selected from the one or moreavailable Kitchen utensils; a required action relating to the one ormore required cooking items; and an instruction relating to the requiredaction; C. displaying, to a user, the plurality of available recipes; D.receiving, from the user, a selected recipe from the plurality ofavailable recipes; E. displaying, to the user, a cooking interfacecomprising the one or more required cooking items associated with acurrent recipe step, wherein each of the one or more required cookingitems is associated with a current state, and wherein each of the one ormore required cooking items is displayed via the cooking interfaceaccording to the respective current state; F. providing, to the user,the instruction associated with the current recipe step; G. a receiving,from the user, via the cooking interface, a user action relating to theone or more required cooking items associated with the current recipestep, wherein the user action comprises one or more of the groupconsisting of: adding a required cookware item to a required kitchenappliance, removing a required cookware item from a required kitchenappliance, adding a required ingredient to a required kitchen appliance,removing a required ingredient from a required kitchen appliance, addinga required ingredient to a required kitchen utensil, removing a requiredingredient from a required kitchen utensil, moving a required kitchenutensil to a required ingredient, adding a required ingredient to arequired cookware item, removing a required ingredient from a requiredcookware item, adding a first required ingredient to one or moreadditional required ingredients, setting a parameter of a requiredkitchen appliance, and setting a parameter of a required kitchenutensil; H. providing, to the user, feedback information relating to theuser action; and I. upon determining, that the user action matches therequired action associated with the current recipe step: updating thecurrent state of the one or more required cooking items associated withthe current step, based on the user action; and upon determining, by thecomputer, that the current recipe step is not a last recipe stepassociated with the selected recipe: updating the current recipe step toa next recipe step associated with the selected recipe; and repeatingsteps E through h wherein the selected recipe is associated with a firstrecipe step that itself is associated with; first required cooking itemscomprising: a first required kitchen appliance associated with aninitial state and a final state, both comprising an active status and alocation within the cooking interface: a first required cookware itemassociated with: an initial state comprising an empty status, a locationwithin the cooking interface that is on or in the first required kitchenappliance: and a final state comprising a full status and the location:and a first required ingredient associated with: an initial statecomprising an uncooked status and an initial location within the cookinginterface: and a final state comprising a cooked status and a finallocation within the cooking interface that is on or in the firstrequired cookware item and that is different than the initial location;a first required action comprising moving the first required ingredientfrom the initial location to the final location; and a first instructionrelating to the first required action, and wherein, when the firstrecipe step is the current recipe step: the first instruction isdisplayed via the cooking interface; until a user action matching thefirst required action is received, each of the first required cookingitems is displayed via the cooking interface according to its initialstate; and upon receiving a user action that matches the first requiredaction each of the first required cooking items is displayed via thecooking interface according to its final state.
 16. A system accordingto claim 15, wherein the user action comprises setting a parameter of arequired kitchen appliance.
 17. A system according to claim 16, whereinthe user action further comprises adding a required ingredient to therequired kitchen appliance.
 18. A system according to claim 17, whereinthe parameter of the required kitchen appliance comprises one or more ofthe group consisting of: a time and a temperature.